System and method for analyzing big data activities

ABSTRACT

A system and method for analyzing big data activities are disclosed. According to one embodiment, a system comprises a distributed file system for the entities and applications, wherein the applications include one or more of script applications, structured query language (SQL) applications, Not Only (NO) SQL applications, stream applications, search applications, and in-memory applications. The system further comprises a data processing platform that gathers, analyzes, and stores data relating to entities and applications. The data processing platform includes an application manager having one or more of a MapReduce Manage, a script applications manager, a structured query language (SQL) applications manager, a Not Only (NO) SQL applications manager, a stream applications manager, a search applications manager, and an in-memory applications manager. The application manager identifies if the applications are one or more of slow-running, failed, killed, unpredictable, and malfunctioning.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/047,554, filed Feb. 18, 2016, which claims priority to U.S.provisional patent application 62/117,902, filed Feb. 18, 2015, entitled“SYSTEM AND METHOD FOR ANALYZING BIG DATA ACTIVITIES,” the entiredisclosures of which are hereby incorporated by reference.

FIELD

The present system and method relate generally to the field ofcomputers, and particularly to a system and method for analyzing bigdata activities.

BACKGROUND

Big data systems are very complex. The productivity of applicationdevelopers and operations staff plummets when they have to constantlytrack many interdependent factors such as application behavior, resourceallocation, data layout, and job scheduling to keep big dataapplications running. Problems associated with the operation of big datasystems becomes hard to identify, diagnose, and fix.

SUMMARY

A system and method for analyzing big data activities are disclosed.According to one embodiment, a system comprises a distributed filesystem for the entities and applications, wherein the applicationsinclude one or more of script applications, structured query language(SQL) applications, Not Only (NO) SQL applications, stream applications,search applications, and in-memory applications. The system furthercomprises a data processing platform that gathers, analyzes, and storesdata relating to entities and applications. The data processing platformincludes an application manager having one or more of a MapReduceManage, a script applications manager, a structured query language (SQL)applications manager, a Not Only (NO) SQL applications manager, a streamapplications manager, a search applications manager, and an in-memoryapplications manager. The application manager identifies if theapplications are one or more of slow-running, failed, killed,unpredictable, and malfunctioning.

The above and other preferred features, including various novel detailsof implementation and combination of elements, will now be moreparticularly described with reference to the accompanying drawings andpointed out in the claims. It will be understood that the particularmethods and apparatuses are shown by way of illustration only and not aslimitations. As will be understood by those skilled in the art, theprinciples and features explained herein may be employed in various andnumerous embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included as part of the presentspecification, illustrate the various embodiments of the presentdisclosed system and method and together with the general descriptiongiven above and the detailed description of the preferred embodimentgiven below serve to explain and teach the principles of the presentdisclosure.

FIG. 1 is an exemplary illustration of applications that the presentsystem and method support, according to one embodiment.

FIG. 2 is an exemplary illustration of the system architecture of thepresent system, according to one embodiment.

FIG. 3 is an exemplary illustration of an exemplary events panel for anapplication, according to one embodiment.

FIG. 4 is an exemplary illustration of a live workflow SLA alert andlive resource wastage alert, according to one embodiment.

FIG. 5 is an exemplary illustration of a user interface for theapplication manager, according to one embodiment.

FIG. 6 is an exemplary illustration of a user interface for resourceusage breakdown, according to one embodiment.

FIG. 7 is an exemplary illustration of a user interface for databreakdown, according to one embodiment.

FIG. 8 is an exemplary illustration of a navigation user interface,according to one embodiment.

FIG. 9 is an exemplary illustration of a user interface for providing anexecution view, according to one embodiment.

FIG. 10 is an exemplary illustration of a user interface for a MapReducetimeline, according to one embodiment.

FIG. 11 is an exemplary illustration of a user interface for a skewview, according to one embodiment.

FIG. 12 is an exemplary illustration of a user interface for providinglog information, according to one embodiment.

FIG. 13 is an exemplary illustration of a user interface for providingconfiguration information, according to one embodiment.

FIG. 14 is an exemplary illustration of a user interface for centralizedworkflow, according to one embodiment.

FIG. 15 is an exemplary illustration of a user interface for an instancecompare, according to one embodiment.

FIG. 16 is an exemplary illustration of a user interface for navigation,according to one embodiment.

FIG. 17 is an exemplary illustration of a user interface for providingan execution view, according to one embodiment.

FIG. 18 is an exemplary illustration of a user interface for providing acentralized applications view, according to one embodiment.

FIG. 19 is an exemplary illustration of a user interface for a resourcemanager, according to one embodiment.

FIG. 20 is an exemplary illustration of a user interface for aleaderboard, according to one embodiment.

FIG. 21 is an exemplary illustration of a service manager, according toone embodiment.

FIG. 22 is an exemplary illustration of a user interface for acentralized inefficiencies display, according to one embodiment.

FIG. 23 is an exemplary illustration of a user interface for a reportingdisplay, according to one embodiment.

It should be noted that the figures are not necessarily drawn to scaleand that elements of structures or functions are generally representedby reference numerals for illustrative purposes throughout the figures.It also should be noted that the figures are only intended to facilitatethe description of the various embodiments described herein. The figuresdo not describe every aspect of the teachings described herein and donot limit the scope of the claims.

DETAILED DESCRIPTION

A system and method for analyzing big data activities are disclosed.According to one embodiment, a system comprises a distributed filesystem for the entities and applications, wherein the applicationsinclude one or more of script applications, structured query language(SQL) applications, Not Only (NO) SQL applications, stream applications,search applications, and in-memory applications. The system furthercomprises a data processing platform that gathers, analyzes, and storesdata relating to entities and applications. The data processing platformincludes an application manager having one or more of a MapReduceManage, a script applications manager, a structured query language (SQL)applications manager, a Not Only (NO) SQL applications manager, a streamapplications manager, a search applications manager, and an in-memoryapplications manager. The application manager identifies if theapplications are one or more of slow-running, failed, killed,unpredictable, and malfunctioning.

In the following description, for purposes of clarity and conciseness ofthe description, not all of the numerous components shown in theschematic are described. The numerous components are shown in thedrawings to provide a person of ordinary skill in the art a thoroughenabling disclosure of the present system and method. The operation ofmany of the components would be understood to one skilled in the art.

Each of the additional features and teachings disclosed herein can beutilized separately or in conjunction with other features and teachingsto provide a detachable frame for a mobile computer. Representativeexamples utilizing many of these additional features and teachings, bothseparately and in combination, are described in further detail withreference to the attached drawings. This detailed description is merelyintended to teach a person of skill in the art further details forpracticing preferred aspects of the present teachings and is notintended to limit the scope of the present disclosure. Therefore,combinations of features disclosed in the following detailed descriptionmay not be necessary to practice the teachings in the broadest sense andare instead taught merely to describe particularly representativeexamples of the present teachings.

Moreover, various features of the representative examples and thedependent claims may be combined in ways that are not specifically andexplicitly enumerated in order to provide additional useful embodimentsof the present teachings. In addition, it is expressly noted that allfeatures disclosed in the description and/or the claims are intended tobe disclosed separately and independently from each other for thepurpose of original disclosure, as well as for the purpose ofrestricting the claimed subject matter independent of the compositionsof the features in the embodiments and/or the claims. It is alsoexpressly noted that all value ranges or indications of groups ofentities disclose every possible intermediate value or intermediateentity for the purpose of original disclosure, as well as for thepurpose of restricting the claimed subject matter. It is also expresslynoted that the dimensions and the shapes of the components shown in thefigures are designed to help understand how the present teachings arepracticed but are not intended to limit the dimensions and the shapesshown in the examples.

The present system and method creates a holistic view of big dataactivities. An operator can quickly understand what is working, getinsights into potential issues proactively, and easily dig deep intowhat is not working or what is not working properly in an integrated andintelligent platform. The present system and method also increases teamand system productivity, empowers developers to become self-servicing,and transforms operations from reactive to proactive.

Whether it is an ad-hoc SQL inquiry in Hive or a repeatedly-run workflowin Spark running either on the cloud or an on-premises cluster,extensive knowledge of the inner workings of big data systems isrequired to obtain the desired application performance and clusterefficiency. The present system and method learns about big data usage,identifies errors, and recommends solutions. Thereby, companies canfocus on their core business issues instead of spending time andresources at optimizing big data applications and their systems.

The present system and method also allows companies to take control ofthe entire big data operation from applications to infrastructure andallow them to focus on making big data work for their business. Thepresent system and method analyzes big data activities from differentdimensions to continuously spot and eliminate problems that lowerreturn-on-investment, impede data democracy, and prevent the companyfrom innovating with big data.

The present disclosure describes an intelligence platform that helpssimplify, optimize and control big data activities. The presentintelligence platform works for big data application engines and can beused for any type of deployment: bare-metal, cloud, or hybrid. Thepresent system and method provides big data analytics to allow companiesand service providers to increase performance of big data systems, forexample, increasing a Hadoop workload by multiple times, while savingtime in finding optimal settings. Typical Hadoop dashboards provideinformation about the system performance but an expert is required toidentify system errors or failures and fix them. The present systemmakes it easy to discover and solve problems by making operationsproactive and users self-servicing.

FIG. 1 shows an exemplary big data system, according to one embodiment.The top part of FIG. 1 shows the breadth of applications that thepresent system and method can support. A program is submitted on a bigdata system to produce desired results. These applications come in awide variety such as MapReduce, Pig, Hive, Tez, HCatalog, HBase,Accumulo, Storm, In-Mem Search, etc. Applications can be submitteddirectly or be submitted through higher-level software such as Tabaleu.It is apparent that the present system and method can cover otherapplication types submitted both directly and indirectly.

FIG. 2 shows the system architecture of the present system, according toone embodiment. The present intelligence platform 200 consists of a core210 and a suite of applications 220. The core 210 is a data processingplatform that constantly gathers, analyzes, and stores data aboutentities and applications of the distributed file system 250 (e.g.,Hadoop distributed file system). The distributed file system 250 runs adata operating system 260 (e.g., YARN) and various applicationstherefrom such as Script applications (e.g., Pig), structured querylanguage (SQL) applications (e.g., Hive, HCatalog), Not Only (NO) SQLapplications (e.g., HBase, Accumulo), stream applications (e.g., Storm),search applications (e.g., SoIr), In-memory applications (e.g., Spark),and other applications (e.g., YARN-ready applications). The results ofthe analysis may be fed back to the distributed file system 250 toincrease the performance of the distributed file system 250. The presentsystem processes this data to power the applications.

The present intelligence platform engine applies several processes tothe data gathered in the core to power events, alerts, root-causeanalysis, recommendations and solutions. Table 1 shows exemplaryfeatures of the present intelligence platform.

TABLE 1 Applications Features 1) Application Manager Includes DAG, KPIsand app details. a) MapReduce Manager Enables live, automatic diagnosisand resolution b) Pig Manager of: c) Hive Manager Slow-runningapplications d) Spark Manager Failed or killed applications e) CustomApplication Manager Unpredictable application behavior Applicationsproducing incorrect results 2) Workflow Manager Includes DAG, KPIs andworkflow details. a) Oozie Workflow Manager Correlates workflows tobusiness processes b) Custom Workflow Manager Enables monitoring andmeeting workflow Service Level Agreements (SLAs) Performs comparison,trending, and reporting of data access, resource usage, configuration,etc., among various runs of a workflow over time 3) Core AppsAutomatically detects, diagnoses, and a) Deep Events recommendssolutions for inefficiencies and errors b) Live Alerts in applicationsand resources c) Info Tags Provides custom alerts to track allactivities, d) Entity Search inefficiencies, and errors in the clusterEnables custom tagging of applications, resources, tables, users, etc.,for reporting, discoverability, and comparison. Enables powerful andeasy search over applications, resources, tables, users, etc. 4) OpsCentral Enables automatic diagnosis and resolution a) ApplicationCentral of inefficient cluster usage b) Resource Central Proactivelyalerts on inefficient or c) Data Central inappropriate use of data andresources by d) User Central applications e) Service CentralChargeback/Showback - Splits usage and cost of f) Inefficiencies Centralthe cluster and resources by user group, g) Reporting application type,etc. Automatically creates leaderboards to identify the mostresource-consuming and resource- wasting applications, users, tables,and queues Correlates changes in application performance with changes indata size, resource contention, and performance degradation of Hadoopservices 5) Data Manager Identifies hot and cold tables a) Top-N listsSummarizes data usage and access patterns b) Pattern Analyzer acrossfiles, tables, queries, columns, and joins c) Auditor Recommends thebest data partitioning and storage layout based on usage Audits dataaccess by applications and users 6) Planner Identify cheapest or fastestinfrastructure for workload Which datasets to move Which ApplicationEngine to choose How to allocate resources When and how to scale-up andscale-down

The present intelligence platform comes with built-in core applicationsthat power smart features and allow for the productive and efficientmanagement of big data systems. Examples of core applications include,but are not limited to, Deep Events, Live Alerts, Info Tags, and EntitySearch.

Deep Events—powered by machine-learning algorithms. The Deep Eventsapplication takes a role of a Hadoop expert to identify errors andinefficiencies in the system automatically. The Deep Events applicationalso provides automatic root-cause analysis and solutions forapplication and system-level problems. The Deep Events application istightly integrated with other applications such as the ApplicationManager, Workflow Manager, Ops Central, and Data Manager.

FIG. 3 shows a screen shot of an exemplary events panel for anapplication, according to one embodiment. The Events Panel may listother events associated with the application to provide analysis andsolutions for the application.

Live Alerts—users can set custom alerts on applications, workflows,users, tables, queues, etc., to proactively find and eliminate errorsand inefficiencies. Live Alerts are Twitter-like streams that bringusers' attention to the most important tasks. FIG. 4 shows a screen shotof an exemplary Live Alert, according to one embodiment. The Live Alertsshows workflow Service Level Agreement (SLA) alert and resource wastage.

Info Tags—the present intelligence platform provides tagging as an easymechanism to improve grouping and discoverability. Teams can tagapplications, workflows, tables, users, and any other entity to be ableto search, group, and report over.

Entity Search—the integrated search engine of the present intelligenceplatform intelligently indexes and associates all cluster activity withapplications, workflows, tables, users, queues, machines, and otherentities. With Entity Search, developers and operations staff can searchfor any entity and get all relevant information for it instantly in asingle screen. For example, searching for a user presents allinformation about that user including applications run, resources used,inefficiencies caused, etc.

The Application Manager provides a comprehensive view into the behaviorof MapReduce, Pig, Hive, Spark, and custom applications. The ApplicationManager is used by Hadoop application owners (e.g., developers, businessanalysts, data scientists) to quickly understand and resolveinefficiencies, bottlenecks, and reasons for application failure.

In addition to applications being submitted directly through Hive, Pig,Spark, etc. the application manager also captures applications that arenot being submitted directly but through a third party program. Forexample, a user who wants to check out graphs using Tableau software maysubmit Hive applications under the cover. The application manager tellsthe user when his/her graph does not show up or takes a long time toload. FIG. 5 shows a screen shot of the application manager, accordingto one embodiment. The application manager shows the events and thestatus (e.g., success, killed), a duration, data I/O, and the number ofresources, the execution view.

The Application Manager includes the following features:

-   -   1) Deep Events—Deep Events automatically identify inefficiencies        and errors. Deep Events also provide automated remedies as well        as guidance on how to resolve the identified application        problems.    -   2) Resource usage breakdown—Resource usage breakdown understands        how and what resources are used by an application. Resource        usage breakdown shows slot usage and map/reduce attempt        successes and failures in an intuitive visual. FIG. 6 shows a        screen shot of an exemplary resource usage breakdown, according        to one embodiment. The number of attempts for map tasks and        reduce tasks are shown in a slot usage graph including a slot        duration.    -   3) Data breakdown—Data breakdown understands how and how much        data is read and written by an application and by every stage in        the application. FIG. 7 shows a screen shot of an exemplary data        breakdown, according to one embodiment. The data associated with        a MapReduce are shown in a tabular format listing the table        names, the amount of I/O data, the number of records, and the        number of tasks.    -   4) Duration breakdown—Understand how time was spent in various        stages and phases of the application execution.    -   5) Navigation—Navigation provides an easy navigation of        applications and their components with drill-down capabilities.        FIG. 8 shows a screen shot of an exemplary navigation, according        to one embodiment.    -   6) Execution view—Execution view provides a graphical view of        application execution. Execution view helps identify bottlenecks        and get detailed information about each stage of the        application. FIG. 9 shows a screen shot of an exemplary        execution view, according to one embodiment.    -   7) MapReduce timeline—MapReduce timeline provides a detailed        view of each map/reduce task along a timeline on the machines        that an application is executed. MapReduce timeline includes        filters to display map, reduce, failed, killed, and successful        tasks. MapReduce timeline helps to understand map, reduces        execution phases, and narrow down problematic tasks and        machines. FIG. 10 shows a screen shot of an exemplary MapReduce        timeline, according to one embodiment.    -   8) Skew view—Skew view shows distribution of time and size of        map and reduce tasks to identify a skew. Skew view also acts as        a filter to narrow down skewed tasks on the MapReduce timeline.        FIG. 11 shows a screen shot of an exemplary skew view, according        to one embodiment.    -   9) Log information—Log information selects an application, a        job, and task-level logs to provide the user with complete        information about an application. FIG. 12 shows a screen shot of        an exemplary log information, according to one embodiment.    -   10) Configuration information—Configuration information provides        a complete list of all configuration settings affecting an        application. FIG. 13 shows a screen shot of an exemplary        configuration information, according to one embodiment.

The Workflow Manager is designed to integrate with popular workflowengines such as Oozie and cron or custom workflow engines. The WorkflowManager provides an intuitive view to understand workflows and itscomponents in a comprehensive manner. The Workflow Manager is used byworkflow (or pipeline) owners to identify and eliminate anomalies,inefficiencies, and bottlenecks in workflow instances, and to guaranteeSLAs. The Workflow Manager includes the following features:

-   -   1) Workflow Central—Workflow Central provides a single view of        defined or ‘tagged’ workflows in the system. Workflow Central        also provides status of how each workflow is performing against        its SLA. FIG. 14 shows a screen shot of an exemplary workflow        central, according to one embodiment.    -   2) Workflow Detail Page—Workflow Detail Page provides a view        into each workflow to provide users a detailed understanding of        the workflow execution. Workflow Detail Page includes:        -   a. Deep Events—Deep Events automatically identify            inefficiencies and errors. Deep Events also provide            automated remedy as well as guidance on how to resolve the            identified problems for workflows. In one embodiment, Deep            Events are the same as in application management.        -   b. Instance Compare—Instance Compare provides an intuitive            comparison graph showing duration, resources used, data            processed, and number of applications across various            instances of the workflow. Instance Compare is particularly            useful to identify anomalies and trends. FIG. 15 shows a            screen shot of an exemplary instance compare, according to            one embodiment.        -   c. Navigation—Navigation provides an easy navigation of the            workflow and its components with drill-down capabilities.            FIG. 16 shows a screen shot of an exemplary navigation,            according to one embodiment.        -   d. Execution view—Execution view provides a graphical view            of workflow execution. Execution view helps to identify            bottlenecks and get detailed information about each stage of            the workflow. FIG. 17 shows a screen shot of an exemplary            execution view, according to one embodiment.        -   e. Resource usage breakdown—Resource usage breakdown            understands how and what resources were used by the            workflow. Resource usage breakdown also shows slot usage,            number of map/reduce task successes and failures in an            intuitive visual. In one embodiment, Resource usage            breakdown is the same as application manager.        -   f. Data breakdown—Data breakdown understands how and how            much data is read and written by the workflow. In one            embodiment, the Data breakdown is the same as application            manager.        -   g. Duration breakdown—Duration breakdown understands how            time is spent in various stages and phases of the workflow            execution.

Ops Central is an extensible application that provides cluster-levelinformation for diagnosing inefficient usage. Ops Central is used byHadoop administrators and operations engineers to ensure optimalresource usage. Ops Central includes the following features:

-   -   1) Deep Events and Live Alerts—Deep Events and Live Alerts        automatically detect inefficiencies with applications, tables,        users, and resources and provide alerts to bring attention to        missing SLAs, applications wasting resources, and more. FIG. 4        shows a screen shot of workflow SLA alert and resource wastage        alerts.    -   2) Application Central—Application Central shows applications        that are running, complete, killed, or failed along with their        KPIs across several clusters on one screen. Application Central        helps to easily understand the workload at any point in time.        Application Central allows for grouping applications by type,        queues, and business units for easy reporting. Application        Central provides the capability to dig into any of these        applications to understand more about the application.        Application Central is useful when debugging or tuning a        particular application of interest. Application Central also        allows operations engineers to prioritize jobs, kill rogue jobs        and eliminate problems before they become a nuisance on the        cluster. FIG. 18 shows a screen shot of an exemplary        applications central, according to one embodiment. For        applications and workflows running today. The number of        successes and killed tasks and the percentage of success rates        are shown on the Application Central. According to one        embodiment, applications may be grouped by the different        operations of a company, for example, financial, advertising,        modeling, and default.    -   3) Resource Manager—Resource Manager shows how queues in the        system are being used at any point in time. Resource Manager        helps to compare current utilization against average and        configured utilization levels to understand which queues are        over and under-utilized. Resource Manager helps system        administrators with resource allocation and capacity planning.        FIG. 19 shows a screen shot of an exemplary resource manager,        according to one embodiment.    -   4) Leaderboard—Leaderboard helps to analyze how the system is        used. Leaderboard can break down cluster usage by users, data,        resources, and queues for workload analysis. Besides showing who        is using the system the most, Leaderboard is useful when trying        to determine how to partition data, how to allocate resources,        and tying system processes to business units. FIG. 20 shows a        screen shot of an exemplary leaderboard, according to one        embodiment.    -   5) Service Manager—Service Manager provides a customizable view        to understand the status of various services running on the big        data platform. Examples of services include, but are not limited        to, Pig, Meta-store, and Hue. FIG. 21 shows a screen shot of an        exemplary service manager, according to one embodiment.    -   6) Inefficiencies Central—Inefficiencies Central provides an        automatically curated list of applications that have caused        inefficiencies in the system. Inefficiencies such as resource        wastage, inefficient join, etc. are captured by Inefficiencies        Central. FIG. 22 shows a screen shot of an exemplary        inefficiencies central, according to one embodiment.    -   7) Reporting—Reporting includes chargeback/showback reports to        understand how each user/department is utilizing the cluster,        their inefficiencies and wastage. Reporting also includes dollar        amounts for each of the above. Reporting also includes cluster        utilization forecasting report that helps in planning upgrades.        FIG. 23 shows a screen shot of an exemplary reporting, according        to one embodiment.

Data Manager is designed to provide data architects a comprehensiveanalysis of data access and usage.

-   -   1) Top-N lists provides customizable top-N lists to show which        tables, columns, joins, and queries are used the most in the        system. Top-N lists also allows users to better partition and        layout their data.    -   2) Pattern analyzer identifies the most important tables and        columns in workloads to help design the right storage strategy.    -   3) Auditor reports data usage and access by applications and        users.

Planner helps developers, operations and business leaders to plan theiractivities to achieve efficiency and productivity. Planner identifiesthe cheapest or fastest infrastructure for a workload. The presentintelligence platform can take any workload (an application or group ofapplications) profile it, and simulate this workload on any type ofserver. The result of the simulation is a graph that compares time andcost across the various server types.

Which datasets to move: the present intelligence platform maintains ahistory of every time a data set or table was touched by an application.This information can be used by companies to help them decide what datato move. For example, if they only wanted recently accessed data tomove, they could see the summary of all such tables very easily in thepresent intelligence platform.

Which Application Engine to choose: similar to which datasets to move,the present intelligence platform can also help to choose an applicationengine that can work best for a particular workload by profiling thatworkload and simulating execution across various engine types.

How to allocate resources: Allocating resources within an organizationis a challenging task. To find the best allocation one needs tounderstand how each department/user group is using the infrastructure.Since the present intelligence platform has information about everyapplication that ran, the present intelligence platform can tellcompanies which department is using what percentage of resources. Thishelps the companies to allocate resources in an intelligent way ratherthan simply guessing or splitting down the middle.

When and how to scale-up and scale-down—the present intelligenceplatform can forecast future usage based on current usage. Therefore,the present intelligence platform can warn the operator ahead of timewhen running out of capacity. Because the present intelligence platformcan also find inefficiencies, the present intelligence platform can alsotell what percentage of the resources is used efficiently versus what iswasted. This can help companies plan whether they should scale up ordown accordingly.

While particular embodiments have been illustrated and described, itwould be obvious to those skilled in the art that various other changesand modifications can be made without departing from the scope of theinvention. It is therefore intended to cover in the appended claims allsuch changes and modifications that are within the use.

We claim:
 1. A system, comprising: a distributed file system for theentities and applications, wherein the applications include one or moreof script applications, structured query language (SQL) applications,Not Only (NO) SQL applications, stream applications, searchapplications, and in-memory applications; and a data processing platformthat gathers, analyzes, and stores data relating to entities andapplications, the data processing platform including, an applicationmanager having one or more of a MapReduce Manage, a script applicationsmanager, a structured query language (SQL) applications manager, a NotOnly (NO) SQL applications manager, a stream applications manager, asearch applications manager, and an in-memory applications manager,wherein the application manager identifies if the applications are oneor more of slow-running, failed, killed, unpredictable, andmalfunctioning.